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Background 

• UAS integration into the National Airspace System 

- In order to comply with 14CFR section 91.113, pilots must be able to 
'see and avoid' nearby traffic that would otherwise create a collision 
hazard 

- Therefore, a Detect and Avoid (DAA) system must be developed to 
compensate for pilots' situated in ground control station (GCS) 

• DAA System must provide information that supports 

pilots' ability to: 

1. Detect potential threats to well clear 

2. Determine appropriate resolutions 

3. Execute maneuvers using the GCS's command and control interface 



Background 


• Precise information requirements have not been 
determined, although studies have started to 
investigate: 

— Friedman-Berg, Rein & Racine, 2014 

• Determined a minimum amount of information (intruder state & 
trend information, and alerting) in a low-fidelity simulation 

- Draper, Pack, Darrah & Moultan, 2014 

• Found additional information (in form of DAA guidance) improved 
pilot performance and was rated as subjectively necessary by 
pilots 

- Fern, et al., 2015 

• Examined impact of traffic display location and DAA guidance tools 
on pilots' performance and subjective experience 



Background 


• Fern et al. (2015) 

- Found that an integrated traffic display (i.e., collocated 
with the GCS navigation and control interfaces) that 
included DAA guidance tools led to: 

• Faster pilot response times 

- Pilots able to implement their 'edits' through the GCS interface more 
quickly to get the aircraft maneuvering 

• Higher subjective ratings 

- Was rated as the best display in supporting pilots' ability to maintain well 
clear 

• Fewer and less severe losses of well clear (Santiago et al. (2015)) 

- However: 

• The integrated, advanced display had multiple DAA guidance tools, 
limiting researchers' ability to localize the source of the improved 
pilot response times (and lower rates of loss of well clear) 


Background 


• Purpose of current study: 

- Decompose the integrated, advanced display from 
Fern et al. (2015) to better understand its results 

• Determine which, if any, display features were most 
responsible for faster pilot response times 


Method 


• Participants 

- 9 active UAS pilots ( M = 46 years of age) 

• Experience: 

- Military UAS (combat & non-combat) = avg. 1182 hours 

- Civilian UAS = avg. 153 hours 

- 1 retired air traffic controller experienced in simulated airspace served as 
confederate 

- 2 general aviation pilots to serve as "pseudopilots," controlling the simulated 
man traffic in the area 


Method 

• Simulation Environment 

- Ground Control Station 

• Vigilant Spirit Control Station (VSCS; Air Force Research Lab) 

- Tactical Situation Display (TSD; traffic information & control 
interfaces) 

- Out-the-window view (synthetic nose camera view with heads-up 
display overlay) 

- Status panels (telemetry, chat window, electronic checklist) 



Method 


• Simulation Environment 

— Traffic and Airspace Simulation 

• Multi Aircraft Control Station (MACS) 

- Controller Display 

» Provided confederate ATC with experimental sector (Oakland Center, 
ZOA 40/41) 

• Consisted of Class A and Class E airspace 

• Wide variety of aircraft types 

- Traffic Generator 

» Injected scripted manned aircraft into 
experimental sector 

» Designed to emulate realistic traffic 
patterns and flows 

- Pseudo-pilot Display 

» Pilot stations allowed 2 confederate 
pilots to manage all simulated manned 
aircraft 




Method 


• Simulation Environment 

— DAA Algorithm 

• Java Architecture for DAA Modeling and Extensibility (JADEM; Santiago et 
al., submitted) 

- Compared intruder's current state to ownship's trajectory to determine if a 
'loss of well clear' would occur 

» Threat levels assigned according to this determination 

- Contained Autoresolver-AD, which supported DAA guidance tools 



Alert Level 

Icon 

HMD at CPA 

ZTHR 

Time to CPA 

Collision Avoidance 


< 0.8 NM 

< 400 FT 

< 40 secs 

Predicted CA Alert 


< 0.8 NM 

< 400 FT 

< 80 secs 

Self Separation 


< 1.2 NM 

< 900 FT 

< 80 secs 

Preventative 


< 2 NM 

< 900 FT 

< 80 secs 

Proximal 


>2 NM 

> 900 FT 

N/A 


Method 


• Experimental Design 

- Display Configuration 

• Dl: Information Only 

• D2: Information + Vector Planner Tool 

• D3: Information + Auto Resolver 

• D4: Information + Vector Planner Tool + Auto Resolver 


Method 

• Experimental Design 

- Display Configuration 

• Dl: Information Only 

— Intruder state and trend information 
» Callsign 

» Position & Directionality 
» Absolute and Relative Altitude 
» Groundspeed 
» Vertical Trend and Rate 
» Horizontal Predictor (30sec dead reckoning) 

- Alerting 

» Corresponding to status as threat to well clear 



Method 


• Experimental Design 

- Display Configuration 

• Dl: Information Only 

• D2: Information + Vector Planner 

- All information from Dl 
— Vector Planner Tool 

» Allowed pilots to propose a heading or altitude prior to 
uploading the change 


» Would reflect predicted safety level of probed maneuvers 
» Integrated with Auto Pilot interface 


Method 


• Experimental Design 

- Display Configuration 

• Dl: Information Only 

• D2: Information + Vector Planner 

• D3: Information + Auto Resolutions 

- All information from Dl 

- Auto Resolutions 

» A single maneuver recommendation provided to pilot at onset 
of a Preventive, Corrective or Warning Alert 

— Integrated with Auto Pilot interface 


Method 


• Experimental Design 

- Display Configuration 

• Dl: Information Only 

• D2: Information + Vector Planner 

• D3: Information + Auto Resolutions 

• D4: Information + Vector Planner + Auto Resolutions 

- All information from Dl 

- Vector Planner Tool (from D2) & Auto Resolutions (from D3) also 
included 

» Roughly same as the 'Integrated, Advanced 7 suite in Fern et al. 
(2015), which was the highest performer 



Displays 


Information Only 








Display Conditions 




D4 







Method 


• Primary Tasks: 

1. Maintain well clear from other aircraft 

• Coordinate with ATC regarding any deviations from filed route and altitude 

• Monitor traffic display to determine when maneuvers may be necessary 

2. Fly pre-filed path as much as practical 

• Only responsible for navigating the aircraft (IFR) 

- Simulated performance specs of MQ-9 Reaper 

• Requires interaction with the GCS and coordination with ATC 

• Secondary tasks: 

1. 'Chat' directed 

• "Mission" Radio Frequency changes 

• Radial and Distance check 

• Nominal information requests (e.g., fuel level remaining) 

2. System Alerts 

• Generator Failure (checklist) 

• Tanker Fleader Overpressure (checklist) 

• Annunciator checks 
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Method 


• Scenarios 

— Pilots flew 2 different routes 

• Both within Class E 

• Each had a dedicated scripted traffic scenario 

- 40 min route 

• 8 encounters with ownship intended to lose well clear 

- Pilots free to maneuver as appropriate 

• Training 

— Initial training just on operating simulated vehicle through 
Vigilant Spirit Control Station interfaces 

- Dedicated training for each display configuration 
immediately prior to experimental trials 

• Order of presentation counterbalanced between participants 



Method 


• Measures 

— Pilot response time metrics adapted from Fern et al. (2015) 

• Provide insight into pilots' ability to respond to threats of well clear in 
timely manner 

- Pilots' response times help define the the minimum operational performance 
standards (MOPS) for expected pilot and system performance 

- Referenced in human models during fast time simulation 

- Metrics derived using a pilot-DAA interaction timeline 


Stage 

Description 

To 

Self separation alert appears on display 

Ti 

Pilot notifies ATC of desire to maneuver 

t 2 

ATC approves pilot request 

t 3 

Pilot initiates an edit using VSCS control interfaces 

T 4a 

Pilots uploads first maneuver to aircraft 

~^4b 

Pilots uploads final maneuver to aircraft 







Stages of Pilot DAA Task 


• Notification Time (T1-T0) 

- Calculated as the time between the appearance of an alert and the beginning of the 
pilot's transmission to ATC 


,.. Notification.... 
Time 

y y 


T 0 

Ti 

T 2 

T 3 

T 4a 

T 4 b 

Traffic Display Alert 
(SS or CA) 

Pilot Notifies 
ATC 

ATC Approval 

Pilot Initiates 
Edit 

Pilot Uploads 
First Edit 

Pilot Uploads 
Final Edit 


Stages of Pilot DAA Task 


• Notification Time 

• Initial Response Time (T3-T0) 

- Calculated as the difference between the appearance of an alert and the pilot's first 
explicit interaction with the vehicle control interfaces 


i 


Initial Response 
Time 


y 


y 


T 0 

Ti 

T 2 

T 3 

T 4a 

T 4 b 

Traffic Display Alert 
(SS or CA) 

Pilot Notifies 
ATC 

ATC Approval 

Pilot Initiates 
Edit 

Pilot Uploads 
First Edit 

Pilot Uploads 
Final Edit 


Stages of Pilot DAA Task 


• Notification Time 

• Initial Response Time 

• Initial Edit Time (T4a -T3) 

- Calculated as the time between the pilot's first explicit interaction with the vehicle 
control interfaces and the pilot's first upload to the aircraft 


y 


Initial Edit 
Time 

* 


T 0 

Ti 

T 2 

T 3 

T 4a 

T 4 b 

Traffic Display Alert 
(SS or CA) 

Pilot Notifies 
ATC 

ATC Approval 

Pilot Initiates 
Edit 

Pilot Uploads 
First Edit 

Pilot Uploads 
Final Edit 


Stages of Pilot DAA Task 


• Notification Time 

• Initial Response Time 

• Initial Edit Time 

• Total Edit Time (T4b - T3) 

- Calculated as the time between the pilot's first explicit interaction with the vehicle 
control interfaces and the pilot's final upload to the aircraft 
• Identical to Initial Edit Time when only one upload is made 


Total Edit Time 

I 


V 


T 0 

Ti 

T 2 

T 3 

T 4a 

T 4 b 

Traffic Display Alert 
(SS or CA) 

Pilot Notifies 
ATC 

ATC Approval 

Pilot Initiates 
Edit 

Pilot Uploads 
First Edit 

Pilot Uploads 
Final Edit 


Stages of Pilot DAA Task 


• Notification Time 

• Initial Response Time 

• Initial Edit Time 

• Total Edit Time 

• Total Response Time (T4b-T0) 

- Calculated as the time between the onset of a self separation alert and the final upload 
to the aircraft 


Total 

i Response Time ; 

y y 


T 0 

Ti 

T 2 

T 3 

T 4a 

T 4 b 

Traffic Display Alert 
(SS or CA) 

Pilot Notifies 
ATC 

ATC Approval 

Pilot Initiates 
Edit 

Pilot Uploads 
First Edit 

Pilot Uploads 
Final Edit 


RESULTS 

RESPONSE TIMES BY DISPLAY CONFIGURATION 
WITHIN-SUBJECTS ANALYSIS OF VARIANCE 


Results 


• Notification Time 

— No significant main effect of display configuration (p>.05) 

• Initial Response Time 

— No significant main effect of display configuration (p>.05) 


Metric 

D1 

D2 

D3 

D4 

Notification Time 

13.68 

(1.63) 

13.61 

(2.45) 

12.01 

(2.40) 

14.89 

(2.82) 

Initial Response Time 

7.83 

(1.49) 

7.14 

(1.14) 

9.45 

(2.28) 

6.87 

(0.94) 








Results 


• Initial Edit Time 

— Significant main effect (p<.001) 

• D3 & D4 had significantly shorter initial edit times than D1 & D2 

• Total Edit Time 

— Significant main effect (p<.05) 

• D4 had significantly shorter total edit times than D2 



Display Configuration 


Results 


• Total Response Times 

— Approached significance (p=.10) 

• D4 led to the shortest total response times, while D2 led to longest 
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D1 D2 D3 D4 

Display Configuration 


Discussion 


• Study was an attempt to better understand 
findings of Fern et al. (2015) 

- Found pilots had fastest response times when using 
an Integrated, Advanced display 

• Current findings suggest the benefits were due to 
the Auto Resolver tool 

— Present in D3 and D4 configuration which: 

• Cut edit times from 46% to 70% 

• Cut total response times by 33% 

- No difference seen for notification and initiate times 


Discussion 


• Source of improvements in Auto Resolver conditions 
likely due to 2 things: 

1. Directive maneuver guidance 

• By providing a single solution Auto Resolver reduced the amount 
of time it took pilots to decide on an appropriate course of action 

- In Dl, pilots had to determine their own resolution entirely, while in D2 
pilots could use the Vector Planner Tool to decide on necessary 
maneuver 

» Resulted in D2 corresponding to longer edit times than Dl 

2. Coupling of Auto Resolutions with Auto Pilot control 

interfaces in Vigilant Spirit 

• By auto-populating the auto pilot control interfaces, pilots were 
not required to make any 'edits' in the case that they were 
comfortable with the automation's recommendation 

• Pilots complied with Auto Resolution's suggestion 70% of the time 


Discussion 


• Pilots found to respond to alerts, overall, in a timely 
manner 

— Total response times were reduced when pilots had Auto 
Resolver available, however the times were not 
significantly different 

- Relative to Fern et al. (2015) pilots responded more 
quickly 

• Across all conditions, total response times were reduced by 50% 
- Likely due to improved alerting criteria and simplified training 

• Emphasizes the impact ground control station 
interfaces can have on pilot response times 

- By integrating tools with auto pilot interfaces we saw 
greater benefits than may otherwise be seen 



Discussion 


• Limitations 

- Small sample size (n=9) reduced ability to find significant differences 

- Response times do not directly equate to safer system performance 

• Pilots can respond quickly but ineffectively 

• Loss of well clear rates and severity of losses of well clear are critical 

- Santiago and Mueller (submitted) found that D4 resulted in the lowest rate of, and least 
severe, losses of well clear 

» D1 and D2 had the highest rates 

• Future Research 

- Look into de-coupling the guidance from the GCS auto-pilot control 
interfaces 

• The coupling is not likely to be a 'minimum requirement' and therefore may 
not provide realistic response times 

- Simulate different forms of DAA guidance 

• "Banding" displays provide constant, explicit feedback 

- Replicate experiment with different GCS interfaces 

• Results therefore not entirely generalizable - GCS control interfaces vary 
wildly between platforms 


